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THE TECHNIQUE OF COMBINING INCOMPLETE JUDG- 
MENTS OF THE RELATIVE POSITIONS OF N 
FACTS MADE BY N JUDGES 

INVESTIGATIONS in the mental and social sciences often need 
to use measurements by relative position or rank or "order of 
merit." In such cases the series of facts to be ranked may not in 
entirety be known to any one judge. One judge may be able to form 
an opinion concerning only facts 1, 2, 3, 4, and 5; another judge 
may be able to rate only facts 1, 2, 6, 7, 8, and 19; a third judge 
may be able to rate only facts 6, 7, and 9. The investigator has then 
the problem of assigning the most probable order of merit to facts 1 
... w, on the basis of a combination of several partial orders ; or of 
assigning a fairly probable order obtainable without undue expense 
of time. 

The technique for handling this problem has not, to the author's 
knowledge, been seriously considered, though, of course, the prin- 
ciples involved would be obvious to anybody familiar with the gen- 
eral theory of measurement by relative position. Since the problem 
will be met very often in applied psychology, I present here a fairly 
satisfactory procedure used in estimating the general intelligence 
imputed to each of thirty-four college freshman by the combined 
judgment of eighteen individuals who knew from 14 to 29 of the 
thirty-four, and rated those whom they knew. We have, that is, as 
our data, the facts of Table I., the ratings being made in accordance 
with the following instructions: 

"Place a cross at the right of the name in the case of the individ- 
uals whom you know, including yourself. Then consider these in- 
dividuals and mark with a 100 the man who, in your opinion, has 
the best intellect. Mark with a 1 the man who, in your opinion, has 
the least intellectual ability. Mark with a 99 the next to the best for 
intellect, and with a 2 the next to the worst. Continue marking 
down from the top or up from the bottom until all the individuals 
that you know are marked. When two individuals seem to be closely 
alike in intellectual ability, give them each two marks (for example, 
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if after marking the first two at the top 100 and 99, the next two seem 
to be closely alike in intellect, mark each of them '98-97'). Grade 
yourself without over-modesty, as impartially as you can. ' ' 

For convenience the first step is to get a rough approximation to 
the true order by inspection, or by computing the median position 
for each individual, or otherwise. It is obvious, for example, from 
inspection that individuals 35, 20, 4, 40, and 2 are high-numbered 
and that individuals 1, 8, and 26 are low-numbered. 

In the case in hand the rough approximation left individuals 19 
and 33 out of account, since they were graded by only one individ- 
ual, and could best be treated separately. The resulting order was 
that of the heading line of Table II. ; and in accordance with it the 
facts of Table I. were rearranged to form Table II. 

The next step is to get a nearer approximation by measuring 
each individual, beginning at the right in comparison with his neigh- 
bor to the left in Table II. This is done by finding the percentage 
of judges (using only those who rated the two individuals in ques- 
tion), who rated the former individual as lower numbered than the 
latter, and computing the amount of difference represented by this 
percentage. The latter computation in this case assumed that the 
variability of the opinions of the judges concerning the ability of an 
individual was approximately that of the so-called normal surface of 
frequency. Thus, 26 is ranked as lower-numbered than 34 by 6, as 
equal by 0, and as higher-numbered by 3. He is, by the assumption, 
.64 times the Median Deviation (P. E.) below 34, in the judgment of 
those who know the two. A table of differences corresponding to 
various divisions of opinion amongst 2, 3, 4, 5, 6, 7, 8, ... n judges 
will be found useful. (Table III. is such a table up to 15 judges.) 
Table IV. gives the result of this series of computations 1 . "Where 
the values are negative, the first rough approximate order evidently 
needs amendment. 

It is amended by comparing the wrongly placed individuals with 
other individuals near whom they seem to belong until, so far as pos- 
sible, a place can be found for them such that they are, in every 
case, so placed as to give + differences or zero differences. In con- 
nection with this amendment, it will be useful to compute also the 
differences from the two or three next neighbors in the new order. 
For, if an individual is rightly placed, he will not only be below the 
one placed above him in the order, but also below the two or three 
next above him. A process of trial and adjustment will be more 
economical than a rigid procedure in this work. The result in the 
case in hand is as shown in Table V. 

i Cases of no division of opinion among the judges, which theoretically 
correspond to an infinite difference, should be assigned values that are reason- 
able in view of the number of judges and the character of the facts judged. 
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The Differences in Terms of the Median Deviation Corresponding to 
Ant Given Division of Opinions Among 2, 3, 4, 5, 6, . . ., 15 Judges 
N = No. of judges; D = Nature of the division among the judges; A = 
Corresponding difference in terms of the Median Deviation (P. E.). 
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"We may now proceed to correct this amended order by assigning 
each individual an amount of difference from individual No. 1, using 
the comparison of him with each of his four neighbors (in the direc- 
tion of No. 1) in Table V. Thus No. 8 is 1.25 from 1, No. 26 is .64 
from 1 by direct comparison, and 2.38 via No. 8, and averages 1.51. 
No. 36 is .89 from 1 by direct comparison, and 2.09 via 8, and 1.51 
via 26, and averages 1.50. No. 15 is theoretically an infinite dis- 
tance from No. 1 by direct comparison. Call this, in view of the 
small number of the judges concerned, 2.00. No. 15 is 1.89 from 1 
via 8 ; 1.77 via 26 ; 2.14 via 36. It therefore is on the average 1.95. 
No. 25 is 2.84 from 1 via 8; 2.51 via 26; 1.12 (1.50-.38) via 36; and 
2.33 via 15. It therefore averages 2.20. The resulting distances of 
each individual from No. 1 are as given in Table VI. 

The use of three indirect measures, rather than one or two or 
four or five, is arbitrary. The estimate could be made more reliable 
by weighting each comparison according to the square-root of the 
number of judges represented by it. The resulting order should not 
without some consideration be used instead of the order derived from 
comparison with next neighbors only; for the next-neighbor com- 
parison, though not utilizing the data fully, is simple and unambig- 
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uous in its meaning. Perhaps some simple system of overweighting 
the next-neighbor comparison would be advisable. On the whole, 
however, we have in this technique an economical and fairly reliable 
means of combining partial orderings. 

When the number of judges represented in a comparison is large 
there is less need of making the final correction by indirect com- 
parisons. When the number of judges is small it may be well to 
compare via five or six near neighbors. 



TABLE VI 
Differences of Each Individual from the Least Intellectual, No. 1, 
Terms of the Median Deviation of the Opinions 



in 



8 

36 
26 
15 
25 
42 
34 

9 
39 
32 
14 
37 

5 
18 
31 
11 



Distance Above 
oal No. 1 Iz 



id] virtual 
21 

22 


Distance Above 
No. 1 
4.02 


1.25 


4.30 


1.50 


10 
30 


4.44 


1.51 


4.54 


1.95 


38 

6 , , 

7 


4.84 


2.20 


5.39 


2.31 


5 61 


2.41 


12 

17 

3 
27 , 

2 

4 
40 , . , , 
35 
20 


5.76 


2.69 


6.08 


2.90 


6.24 


2.94 


6 27 


3.13 


7 30 


3.55 


7 48 


3.76 


7.51 


3.84 


8.52 


3.98 


8.58 
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Columbia University. 


E. L. Thorndikb 



ETHICS AS SCIENCE AND AS ART 



A LMOST any undertaking may be and has been described as a 
-£j- science, just as almost any one may and does call himself a 
professor. 

When I propose to consider ethics as a science it is in the narrowed 
sense of "ordered knowledge of natural phenomena and of the rela- 
tions between them." I have in mind also (a) the saying of Svante 
Arrhenius that "science implies measurement," (5) the practise of 
experimental research instituted by Francis Bacon, and (c) Galileo's 



